Once I prepare all the transformations and joins, I have to schedule this job for every day to run with different set of files. Can anyone let me know how can I achieve this ?
Can you say a little more about what you want to do?
In general, Dremio applies the joins and transformations that you have defined in a virtual dataset at query time.
Once all the joins and transformations completed, now i want to run the same joins every day with different set of files i., I generally want to schedule a job to run every day
If you simply add new data to the source, the next query will apply the joins and transformations to all the data, including the new records. There’s no need to schedule that work to be done. It sounds a little bit like you’re thinking of Dremio as an ETL tool. It isn’t really designed for that. Instead, the work of performing joins and transformations is applied at query time.
Does that make sense?