Hello,
I have a question regarding master coordinator capacity with regards to metadata. Is there any documentation on the limits on how many data lakes and external data sources we can create on a single dremio deployment?
What are the best practices? Is it better to have a single data lake with a lot of sub folders or is it better to create multiple data lakes?
Thank you, Igor
@igorsechyn If this is a file system source then probably creating one source with the top level folder and the create spaces for different VDS would be fine. If there are many different sources then multiple sources with different root paths should be fine but creating too many like 20+ might start introducing some heap memory allocations. Also depends on how powerful in terms of CPU the coordinator is. Best to start somewhere say 20 sources and then monitor the GC logs for any Full GC Pauses