Apply format to source folders with code

Hi at all,
sometimes happends that the hdfs folders loses their format (after problems of reachability of hdfs) and then is impossible to query the relatives pds. I have set the source option “Remove dataset definitions if underlying data is unavailable.” to not. To re-format all the folder we have to spent much time but i don’t find a way to do this via code. Is it possible to use code to format source tables?



In the source settings, under metadata you can check “Automatically format files into physical datasets when users issue queries.”

This option might not work well if you have CSV files as it would apply default formatting options like comma as the field delimiter for example. If you are only dealing with JSON or PARQUET then yes it should help


You can use our REST API to promote files to datasets.