Just one FIeld not mapped in integration with mongodb

I have a field ‘acceptedAt’ in collection ‘users-challenges’ in MongoDB, but just this field not showing in my Dremio, all others field is showing.

Fields from users-challenges collection:


Field that isn’t showing:

image


One field that is showing correctly:

image

@lucasft What type of column is that? or does it only appear in a few documents?

@balaji.ramaswamy In mongoDB is type Date and appear vast majority documents.

@lucasft It could be possible alter pds did not sample this file for schema, can you do `select * from " and click run (not preview) and see if the column now shows up?

@balaji.ramaswamy I clicked in run, but don’t appear.

@lucasft Can you please send us the profile of the “select * from the collection” run?

I got the same problem. In onemongoDB collection in Dremio several fields are missing. Every dataset has these fields, no mixed types for the missing fields…?

@WillBe

Wondering if your UI select is getting truncated, so all documents have the exact same number of fields and each field has the same exact data type on every document?

I’ve checked it and the number of fields in the collection differ. Isn’t Dremio able to handle that?

@WillBe Dremio (not for Parquet, ORC, AVRO) for MongoDB samples files for schema and if you have a column that is not part of those sampled files, you will not see them initially but you should be able to query the file that has that column ot query the entire collection and Dremio should do a schema learning

Dremio has internal schema via SQL where you can define your PDS, can you please try that approach, see documentation below

https://docs.dremio.com/software/sql-reference/sql-commands/tables/#alter-table

Dear Balaji, the behaviour of my collection was: Dremio ignored all the fields that had a null value. I had to fill in a value and do
alter table “<my_context>.table1” forget metadata
alter table “<my_context>.table1” refresh metadata
then the fields appeared.
As soon I set the fields to null and refresh the schema they dissappear again.
So I keep the fields filled in one dataset and everything is fine.
But I’ll try your solutione too and keep you informed if it works or not.
Many thanks for your support!