Manual Dataset Discovery of Glue Catalog

Dremio University
1

Hi!

Dremio OSS UI seems to only provide minute-level dataset discovery.

In my use case, I need to trigger dataset discovery more frequent than what the Dremio OSS UI provides. Ideally, I want to be able to trigger the dataset discovery manually so I can detect new tables as soon as it’s created.

How would I go about doing that?

Thank you!

2

In the source code, the minimal dataset discovery seems to be 1 minute.
Is there no Rest API to force a manual dataset discovery?

3

@kyleahn

You can use the below command

https://docs.dremio.com/software/sql-reference/sql-commands/datasets/#managing-physical-datasets