Create dataset from glob or regex pattern

1

Continuing the discussion from Exclude certain subfolders when creating a dataset:

Is there any way to promote a dataset by using a glob pattern to include/exclude specific files/folders?

If not, a couple of solutions that might work could be:

  • Option to specify an include or exclude regex pattern when promoting a dataset
  • Option to have Dremio respect a .git_ignore file

I have several different scenarios where I am unable to separate mixed file types without investing an unreasonable amount of time and effort. This limitation is the single factor preventing my company from being able to use Dremio as our real-time analytics and ETL solution.

2

@jamesdouglas Dremio should ignore file names starting with a “.”, is Dremio complaining about a .git_ignore file?

Right now the feature to exclude certain files from a folder is not there