Elasticsearch columns getting mixed up in Dremio when index updates

1

I have an Elasticsearch database sucking in data from a Trac ticket system once an hour. I set up a Dremio connection to this ES database and enabled raw reflections.

After the next hour has gone by and new data is loaded, the columns get all mixed up.

Before:

image
image.png942×529 22.9 KB
(redacted some data)

After:

image
image.png754×711 37.3 KB

Also interesting that the id column went from being recognized as a string instead of a number. Indeed, when I try to query SELECT id FROM <virtual dataset>, I get an Unexpected error occurred. Looking at the job report, here was the error:

Error
      AssertionError: Type mismatch:
rowtype of new rel:
RecordType(VARCHAR(65536) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary" 
id) NOT NULL
rowtype of set:
RecordType(BIGINT id) NOT NULL

Any idea what is happening here? I can provide query profiles etc. One of the reasons my company wants to deploy Dremio is to be able to work with our Elasticsearch data as new documents are added and schemas are allowed to change, this is definitely a breaking issue.

2

I’ll let others comment, but first - what version of ES are you on?

3 
{
  "name" : "lgb-es-01",
  "cluster_name" : "my_cluster",
  "cluster_uuid" : "Szuw6f-XTSCMvXZ5CAr_2A",
  "version" : {
    "number" : "5.6.5",
    "build_hash" : "6a37571",
    "build_date" : "2017-12-04T07:50:10.466Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.1"
  },
  "tagline" : "You Know, for Search"
}