Field [XXX] has incompatible types in file and table

1

Hi, I’m trying to understand an error I get when I point Dremio to a dataset.

Field [PAYLOAD] has incompatible types in file and table

In the physical data set the field “PAYLOAD” it is always an object, but frequently has a few different shapes. It is logging an ElasticSearch query and so as the query changed so is the object. Sometimes there is a {“term”:…} sometimes a {“match_phrase_prefix”} or sometimes the bool.must field is an empty array.

Can Dremio make sense of that all and allow me to present these as objects or does every object in that field have to have the same schema?

Field [PAYLOAD] has incompatible types in file and table. Type in fileschema: [PAYLOAD: STRUCT<_source: LIST<data: VARCHAR>, from: BIGINT, query: STRUCT<bool: STRUCT<must: LIST<data: STRUCT<match_phrase_prefix: STRUCT<propertyAddress_tx: VARCHAR>, term: STRUCT<firstOccurrence_fg: BOOLEAN>>>>>, size: BIGINT, sort: LIST<data: STRUCT<transaction_id.keyword: VARCHAR>>>], type in tableschema: [payload: STRUCT<_source: LIST<data: VARCHAR>, from: BIGINT, query: STRUCT<bool: STRUCT<must: LIST<data: INTEGER>>>, size: BIGINT, sort: LIST<data: STRUCT<project_id.keyword: VARCHAR, transaction_id.keyword: VARCHAR>>>].Show less

2

@ra0011

Is this on an elasticsearch source, can you please share the query profile with us?

3

@balaji.ramaswamy I believe I resolved my issue.

The data source is AWS glue, and it was trying to read a field that had the JSON ES query as the value for one of the columns.

The field was being stored as a STRUCT, but when I changed my logging to log it as a string Dremio seems to read it fine and still allows the users to extract field from it.

I cannot show the old query profile because I have been editing the source physical data set as part of my development work to solve this issue.