(Dremio 1.08 / MongoDB 3.4.6) - datasource refresh

1

(Dremio 1.08 / MongoDB 3.4.6)

I have added a datasource (MongoDB) to Dremio. Dremio is connecting to the MongoDB default database “test” which I had created two collections in it and it shows up in Dremio UI. I have added a new collection in MongoDB later, but the new collection is not showing up in the Dremio UI under my datasource(MongoDB).

Is there a configuration parameter for refreshing a datasource in Dremio?

Thanks,
Kaveh

2

Never mind, I found it :slight_smile:

1 Like
3

Could please someone clarify where this “configuration parameter” is?
I have the same problem. Dremio 2.0.1/MongoDB 3.4.9.
Should I use the Refresh Policy parameters in the source settings? How can I refresh the source on demand to get the new Mongo collections shown?

4

Have a look here: https://docs.dremio.com/sql-reference/sql-commands/datasets.html

In addition, metadata is refreshed automatically based on a selchedule. You can look at the data source configurations screen to see the options there. More info here: https://docs.dremio.com/advanced-administration/metadata-caching.html

5

Thanks for the link. What is the recommended way to run these SQL requests? Via REST API?

6

You could do through the SQL console or REST or J/ODBC if this is a one time operation. If this is a regular thing you are scripting then through REST or adjust the config settings to be more aggressive.

7

REST API is probably a good enough method for me.
What is SQL console? Is it a part of Dremio Web client?

8

At the top of any dataset, or click the New Query button at the top:

image
image.png2732×898 217 KB

9

Oh, I did not know you can run arbitrary SQLs there.
The type information was fixed after FORGET METADATA + REFRESH METADATA (just refresh had not worked).
Thanks for you help!

1 Like