API: Invalid Raw Reflection, Must Have at least one DIsplay Field

1

HI, I was trying to create a Raw reflection on a Promoted PDS using dremio’s api.

I promote the file to a PDS, I get this response:

{'entityType': 'dataset',
 'id': 'AAAAAAAAA-AAAAAAAAA-AAAAAA-AAAAAAA',
 'type': 'PHYSICAL_DATASET',
 'path': ['DremioS3Source',
  'JJJJJJJ',
  'ZZZZZZZ',
  'XXXX',
  'YYYYY'],
 'createdAt': '2019-07-19T20:44:52.927Z',
 'tag': '0',
 'format': {'type': 'Parquet',
  'ctime': 0,
  'isFolder': True,
  'location': '/JJJJJJJ/ZZZZZZZ/XXXX/YYYYY',
  'autoCorrectCorruptDates': True},
 'approximateStatisticsAllowed': False,
 'fields': [{'name': 'a', 'type': {'name': 'VARCHAR'}},
  {'name': 'b', 'type': {'name': 'VARCHAR'}},
  {'name': 'c', 'type': {'name': 'TIMESTAMP'}},
  {'name': 'd', 'type': {'name': 'TIMESTAMP'}},
  {'name': 'e', 'type': {'name': 'VARCHAR'}},
  {'name': 'f', 'type': {'name': 'VARCHAR'}},
  {'name': 'g', 'type': {'name': 'VARCHAR'}},
  {'name': 'h', 'type': {'name': 'INTEGER'}},
  {'name': 'i', 'type': {'name': 'TIMESTAMP'}}]}

Then I pass the body of data below to create the reflection:

{
  "entityType": "reflection", 
  "name": 'YYYYYY-raw', 
  "enabled": True, 
  "type": 'RAW', 
  "datasetId": 'AAAAAAAAA-AAAAAAAAA-AAAAAA-AAAAAAA',
  "partitionDistributionStrategy": 'CONSOLIDATED' }

Then I get the following error:

{'errorMessage': 'Invalid reflection: raw reflection must have at least one display field configured',
 'context': [],
 'moreInfo': ''}

In the Documentation you say that the ‘displayFields’ is optional, which is not the case, but ok… not a big deal.

So then, I thought about passing the keys I got from the response as values on the reflection creation.

{
  "entityType": "reflection", 
  "name": 'YYYYYY-raw', 
  "enabled": True, 
  "type": 'RAW', 
  "datasetId": 'AAAAAAAAA-AAAAAAAAA-AAAAAA-AAAAAAA',
  "displayFields": pds_creation_response['fields'],
  "partitionDistributionStrategy": 'CONSOLIDATED' }

Then I got the following error:

'Unrecognized field "type" (class com.dremio.service.reflection.proto.ReflectionField), not marked as ignorable (one known property: "name"])
 at [Source: (org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$UnCloseableInputStream); line: 1, column: 187] (through reference chain: com.dremio.dac.api.Reflection["displayFields"]->java.util.ArrayList[0]->com.dremio.service.reflection.proto.ReflectionField["type"])'

So, on what exact syntax should I pass in “displayFields”?
Would I need make a clumsy for loop to extract just the names of the fields?
Is there another way to do it?

2

The payload should look like this:

{
  "type":"RAW",
  "name":"Raw Reflection",
  "enabled":true,
  "partitionDistributionStrategy":"CONSOLIDATED",
  "displayFields": [
    {"name":"IncidntNum"}
  ],
  "datasetId":"d156ccaf-d0eb-4693-b9db-51d2f8af44bf"
}

It looks like you are adding a type field to each field entry.

1 Like