For creating virtual datasets,We are doing aggregations,joins and calculations,Here my question is that whether transformations which I have did,will it stored as jobs anywhere.So that,we can do some modifications in jobs in future based on the requirement changes
Hi,
You can view the history by editing a virtual data set. Hover over the row for the virtual dataset, click on the gear icon and choose Edit:
This will show you the full SQL that was generated as well as show you the transformation history with all the steps you made before saving the virtual dataset. These are represented as dots on the right hand side:
You can hover over each dot to get a short description of what action you took.
You can go back in the history by clicking on the dots. You can make changes, however you will lose any steps that happened after the step you are editing, basically creating a new branch. You can then choose to Save the changes or do a Save As to create a new virtual dataset.
If you want to have reusable steps that can be edited in the future, it might be better to create virtual datasets for each step and then edit them and change the FROM part of the SQL.
More examples of how to use virtual datasets for curation are included in this tutorial, which may be useful:
Hi ,
Thanks for the reply, it’s really helped me a lot. Is Data Graph option available for the community version of Dremio?
Hi Senthil - Dremio’s data lineage capabilities, including visualization of our data graph, are only available in the Enterprise Edition.
You can request access to the Enterprise Edition on our website: https://www.dremio.com/lp/enterprise-edition/