Create Historical Data in Dremio

1

Hi, i’m new on Dremio and need some help.
I have a dataset but the information changes every day, I need a query wich I can “collect” the data every day and save in a new VDS, only need maximum 4 days of historical data.
Example:

data_information day1
union all
data_information day2
union all
data_information day3
union all
data_information day4

Exist any way to create a new VDS like that?

2

@caz_chelo What is the data set partitioned on?

3

@caz_chelo The way I’ve done this is to place each day files in its own folder structure, when I deal with csv/txt

In S3 for example, imagine the ‘folder structure’ is as follows - I hold data going back a year by day.

root/20211123/mydata.txt
root/20211122/mydata.txt
root/20211121/mydata.txt

etc.

I then place a PDS on root

Select * from the PDS will give extra column names of dir*

I create a VDS selecting from the PDS and code a conversion to date of the ‘dir’ column to a new column called FileDate

I can then select from that where where the FileDate is > ‘today - 4’

I can then retain data OLDER than 4 days but mask it out.

1 Like