A PDS on a folder (Azure Datalake gen2) automatically adds the dir0, dir1, dir2 to the PDS, but I also need the filename included in the PDS, to extract very important information from the filename. I’m dealing with lots of files and to create a folder for each file is not really an option. Is there a way to do this?
I really need to be able to get the filename in a PDS. This is a show stopper for me if I cannot get the filename.
On the source, did you try promiting only the last directory only as a dataset
Eg: if path is ADLS.Dir1.Dir2.Dir3.file1
Promote Dir3 as a dataset and see if that works for you.
@Venugopal_Menda, thanks for the suggestion, but unfortunately that did not work.
Our folder structure looks like this:
{root}/{client}/{type}/{subtype}/{version}/{yyyy}/{MM}/{dd}/{dbname}{valuationdate}{timestamp}.gz
If I create the PDS on the {version} folder, in the result dir0 = {yyyy}, dir1 = {MM} and dir2 = {dd}. That is fine and I want that, but I must also be able to get the full filename, ie {dbname}{valuationdate}{timestamp}, to extract the dbname, valuationdate and the timestamp.