Can anyone recommend the best way to get validation errors back from Dremio when they fail on particular files (or even rows)? For example, I am noticing that if I specify a format with a windows return character line delimeter but then upload a file with unix line delimeters the dremio UI returns to me
“Only one data line detected. Please consider changing line delimiter.”, however i cannot find any way to return this validation via the REST api, and it seems to stop importing data from that point forward without any notification.
Some other examples of validations it may be useful to see:
- When files with different schemas don’t conform to the same directory structure as other files. I notice that if I put a file in the outmost directory structure, the format page in dremio tries to change the format to that file which takes presedence. I think this also silently fails until you go into the format page and notice its trying to use the new schema.
- When files have a different schema altogether, or by one or more columns, but have the same directory structure, the differing columns are automatically added to the format, and the other columns are nullified (this is good behavior, however it would be nice to get a validation error form somewhere about the differing headers).