I have a field ‘acceptedAt’ in collection ‘users-challenges’ in MongoDB, but just this field not showing in my Dremio, all others field is showing.
@lucasft What type of column is that? or does it only appear in a few documents?
@balaji.ramaswamy In mongoDB is type Date and appear vast majority documents.
@lucasft It could be possible alter pds did not sample this file for schema, can you do `select * from " and click run (not preview) and see if the column now shows up?
@balaji.ramaswamy I clicked in run, but don’t appear.
@lucasft Can you please send us the profile of the “select * from the collection” run?
I got the same problem. In onemongoDB collection in Dremio several fields are missing. Every dataset has these fields, no mixed types for the missing fields…?
Wondering if your UI select is getting truncated, so all documents have the exact same number of fields and each field has the same exact data type on every document?
I’ve checked it and the number of fields in the collection differ. Isn’t Dremio able to handle that?
@WillBe Dremio (not for Parquet, ORC, AVRO) for MongoDB samples files for schema and if you have a column that is not part of those sampled files, you will not see them initially but you should be able to query the file that has that column ot query the entire collection and Dremio should do a schema learning
Dremio has internal schema via SQL where you can define your PDS, can you please try that approach, see documentation below
https://docs.dremio.com/software/sql-reference/sql-commands/tables/#alter-table
Dear Balaji, the behaviour of my collection was: Dremio ignored all the fields that had a null value. I had to fill in a value and do
alter table “<my_context>.table1” forget metadata
alter table “<my_context>.table1” refresh metadata
then the fields appeared.
As soon I set the fields to null and refresh the schema they dissappear again.
So I keep the fields filled in one dataset and everything is fine.
But I’ll try your solutione too and keep you informed if it works or not.
Many thanks for your support!
