Dremio timeout connecting mongo

1

Hello we find another bug in Dremio, we have a mongo collection with 49232729 documents, when try run a query we receive Timeout while receiving message error

debuggin Dremio in MongoTable in the method

   private long getTotalDocuments() {
        MongoDatabase db = this.manager.getMetadataClient().getDatabase(this.collection.getDatabase());// 136
        com.mongodb.client.MongoCollection<Document> docs = db.getCollection(this.collection.getCollection());// 137
        return docs.countDocuments();// 138
    }

are calling countDocuments which can take too many time, instead must be use estimatedDocumentCount

2

We had a similar issue when defining datasets on big mongo collections. In our case the issue was that the initial metadata refresh timed out.

It is not super obvious based on the parameter name, but I found that increasing Auth Timeout in the Advanced Options on the mongo source to a sufficiently high value at least allowed us to successfully define the datasets.

3

which version of Dremio are you using? in 25.0 works well, before versions has errors, I have a patch for past versions

4

Just upgraded to 25.0.0 on Friday. I haven’t tested yet if it makes a difference for the metadata refresh issue