Hello, I am trying to figure out what would be the best way to handle configuration management with Dremio and if there is any support in the tool itself.
Considering that most companies are the situation where they have multiple environments, with at least dev and prod. Usually they create VDSs and Reflections in development using fake data, and after validation they are passed selectively to production. It would be nice some process to avoid any potential human error.
@enrique.castillo I am working on exactly this type of tool. I have the basics working that allows pulling the config from one Dremio instance and pushing to another. The pulled config creates SQL files in folders that match the structure in Dremio.
My intent is to create a tool that would allow using a source control to track Dremio changes.
Once I have it ready to share I will post it here.
@mlilius thank you, I am looking forward. There is already already an old project for similar purpose, but it does not fill all my needs. https://github.com/christyharagan/dremio-cluster-cloner
@enrique.castillo I am close to releasing the first version. In my setup I am using MYSQL and AWS S3 so those are the data sources I have tested with. What kind of sources are you using?
I did see that other project and tried it out. The problem I had with that in my environment is it is not smart enough to know which VDSes rely on other VDSes. In my tool I am extracting which tables are used for each VDS and setting them up in the right order.